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Education Leadership Data Analytics (ELDA): 
A White Paper Report on the 2018 ELDA Summit 


Abstract: 

Education Leadership Data Analytics (ELDA) is an emerging domain that is centered at the 
intersection of education leadership, the use of evidence-based improvement cycles in schools to 
promote instructional improvement, and education data science. ELDA practitioners work 
collaboratively with school and district leaders and teachers to analyze, pattern, and visualize 
previously unknown patterns and information from the vast sets of data collected by schooling 
organizations, and then integrate findings in easy to understand language and digital tools into 
collaborative and community-building evidence-based improvement cycles with stakeholders. In 
June of 2018, over 100 participants gathered for the Education Leadership Data Analytics Summit 
at Teachers College, Columbia University in New York City, including researchers, practitioners, 
policymakers, and funders. This report provides a summary of the central issues, themes, and 
recommendations for the future of the field that emerged from the discussions at the ELDA 
Summit event. These issues include building capacity in the field through incentivizing researcher 
practitioner partnerships, and providing conference and networking opportunities, professional 
development, certification, and ultimately degree programs to train ELDA researchers and 
practitioners. Additionally, a central focus of the ELDA field is equity, data security and privacy, 
in concert with open and FAIR data standards to develop and share de-identified data and tools 
across contexts. We conclude the report with a blueprint of possible skills and competencies 
needed for ELDA practitioner training and professional development and _ provide 
recommendations for next steps to help grow the field. 


Keywords: education leadership, evidence-based improvement cycles, education data science, 
big data, data analytics, data driven decision-making, evidence use. 


Purpose and Executive Summary: 

The purpose of this report is to provide a summary white paper report on the outcomes from the 
Education Leadership Data Analytics Summit 2018 at Teachers College, Columbia University. 
Education Leadership Data Analytics (ELDA) is an emerging domain that intersects the three areas 
of education leadership, data and evidence use in schools, and data analytics and data science. To 
learn more about the current conversation and needs in ELDA nationally and globally, and help to 
build capacity throughout the current and emerging research and practice networks in ELDA, we 
brought together over one hundred participants for the Education Leadership Data Analytics 
Summit 2018 at Teachers College, Columbia University in New York City on June 7 and 8, 2018. 
Through this event, researchers, practitioners, and policymakers came together to discuss and learn 
more about ELDA, and we gathered their interactions and responses as participants attended poster 
sessions, panel talks, and an interactive event at the Smith Learning Theater at Teachers College 
(TC). 


In this document, we report on this effort. The gathering yielded multiple exciting areas for the 
future of the field of ELDA. 
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First, the domain and market is ripe for more capacity building offerings for teachers, leaders, 
central office staff, and researchers throughout education. Current offerings do not address all three 
sectors of ELDA of Education Leadership, Evidence-based improvement cycles, and Data 
Science. Additionally, offerings that do exist in the market are not accessible and do not address 
the needs of the market, such as providing professional development and continuing education 
credit (CEU) style non-credit courses to current school and district leaders on ELDA in an 
accessible format. Currently, no institutions offer a masters degree in ELDA. However, there is a 
central job in education of people in schools, districts, states, and nations who “do data” through 
processing and analyzing school data and then visualizing and working with the educators in their 
system to leverage that data for instructional improvement. These education data practitioners 
currently come to their jobs through completely idiosyncratic means, as there is currently no 
formalized degree or certification path for this career. Yet, every district, state, and nation has 
someone doing this job, without a network of support and capacity building. 


Second, the researchers and practitioners in the nascent field of ELDA recognize that there are 
strong opportunities to build more capacity at the overlaps of each of the three domains of ELDA. 
Current work is mostly siloed in the individual domains of education leadership, evidence-based 
improvement cycles, and education data science. While there is a strong literature and practice 
base at the intersection of education leadership and evidence-based improvement cycles, there is 
a strong need for more research, training, and capacity building at the intersections of education 
leadership and education data science as well as evidence-based improvement cycles and 
education data science. Indeed, a central need identified in the ELDA domain is a priority on work 
at the intersection of all three domains, in which the new methods and techniques to visualize, 
pattern, understand, and communicate data and evidence to leaders and stakeholders from the data 
science domains is brought together with the current research and practice of education leadership 
and evidence-based improvement cycles. 


Third, participants throughout the gathering identified equity in education as a central priority for 
ELDA. As the techniques continue to improve around patterning and understanding the large 
amounts of data collected in schools, teachers and school and district leaders have the opportunity 
to leverage this data to focus on instructional and system improvement for students and 
communities that have historically been underserved. The opportunity to understand 
individualized and personalized educational trajectories and interventions for students provides a 
unique opportunity to address the individual needs of students through applying the limited 
resources of schools and districts to those specific needs in a timely manner. Through evidence- 
based improvement cycles, teachers and leaders can work together to build capacity throughout 
their organization to leverage these new types of data and analytics as a means to build 
collaboration, trust, and capacity to improve instruction for each student, and across the 
organization. 


Fourth, issues of data privacy as well as the FAIR and open data and algorithm standards were a 
central concern for ELDA. Throughout the work of ELDA, student data must be kept secure and 
private. Protecting data confidentiality and privacy is a core competency within ELDA. Yet, for 
the field to develop, de-identified and anonymized datasets, case studies, and research sites must 
be developed and published. To aid in the development of the field, these types of data should 
follow the FAIR data standards of Findable, Accessible, Interoperable, and Reusable. 
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Additionally, as algorithms are further developed around this data that can either make 
recommendations on students, or decisions on students, these algorithms must be open and 
accessible. As the bulk of this work is paid for by the taxpayer, open source algorithms are the 
only ethical means to allow school stakeholders to audit and understand how the data are being 
used, and why a school may select specific recommendations and decisions for students and 
teachers. 


Fifth, evidence from the ELDA summit points to a central need to build capacity, tools, datasets, 
and networks of researchers and practitioners. A central priority identified for the ELDA field is 
to build training programs in ELDA, along with digital tools that can first be tested and validated 
and then shared across the field to help provide the infrastructure needed for ELDA to serve the 
field and positively impact schooling organizations and ultimately instructional improvement. 


In this report we first detail the landscape and domain background for Education Leadership Data 
Analytics. We then overview and summarize the outcomes from the ELDA Summit. We conclude 
with an overview of the skills and competencies identified as needed in ELDA for four main roles 
in schools and districts, and provide additional details around these five main recommendations of 
this report. 
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The Domain Landscape of Education Leadership Data Analytics (ELDA) 


Contextual Overview and Rationale 

Across school districts in the US and globally, schools are inundated with increasing amounts of 
data (Bowers, Shoho, & Barnett, 2014; Halverson, 2014; Mandinach, Friedman, & Gummer, 2015; 
Wayman, Shaw, & Cho, 2017). Ever more data flows into schools from accountability systems 
and testing, to more traditional data such as teacher-assigned grades (Bowers, 2011; Brookhart et 
al., 2016), discipline reports, and more recently teacher and principal performance evaluation 
systems (Brocato, Willis, & Dechert, 2014), as well as the growing domain of personalized 
learning (Agasisti & Bowers, 2017; Bowers, 2016; Halverson et al., 2015; Krumm, Means, & 
Bienkowski, 2018; Piety, Hickey, & Bishop, 2014). Educators, and especially school and district 
leaders as well as district central offices and state departments of education, are urged to take 
advantage of these vast sets of data to help drive decisions to allocate the limited resources of 
schools to specific student needs with greater accuracy, efficiency and effectiveness (Bowers, 
2008, 2017; Coburn & Turner, 2011; Farley-Ripple & Buttram, 2015; Honig & Venkateswaran, 
2012; Jimerson & Childs, 2015; Marsh & Farrell, 2015; Schildkamp, Karbautzki, & Vanhoof, 
2014; Schildkamp, Poortman, Luyten, & Ebbeler, 2017; Schildkamp, Poortman, & Handelzalts, 
2016; Spillane, 2012; Supovitz & Morrison, 2015). Yet, currently for school district and state-level 
personnel who primarily work with data (“data strategists”), their daily work consists primarily of 
data reporting for policy compliance (Echeverria et al., 2018; Piety, 2013; Piety et al., 2014; Piety 
& Pea, 2018). One reason for this compliance focus around data is that nationally, district data 
strategists have few to no options to acquire training and certification around data use, data 
integration and management, “e-administration”, data mining, coding, statistics, and most 
importantly, how to work with management to help leaders build capacity within organizations for 
evidence-based improvement cycles through conversations using data and evidence (Agasisti & 
Bowers, 2017; Bowers, 2017). This is despite the strong growth in the last decade in industry, 
technology and the sciences in big data analytics and the emerging domain of data science (Cope 
& Kalantzis, 2016; Gandomi & Haider, 2015; Schutt & O'Neil, 2013). 


Almost 50 years ago, in 1970, in talking about the use of data for educational leadership, Farmer 
predicted that “the greatest impact of the quantitative approach” will not be on problem solving, 
but rather on “problem formulation: the way managers think about their problems—how they size 
them up, bring new insights to bear on them, relate them to other problems, communicate with 
other people about them, and gather information for analyzing them” (p. 21). Since then, 
quantitative methods have become integral to school leadership and management practices as well 
as in management education (Agasisti & Bowers, 2017; Bowers, 2017). This work draws from the 
work of Bruno and Fox (1973) in an extensive report commissioned by the University Council for 
Educational Administration (UCEA) on “Quantitative Analysis in Educational Administrator 
Preparation Programs.” In their report, the authors reviewed and synthesized the literature and 
analyzed the “current thought and practice” of their time that was “relevant to the inclusion of 
quantitative analysis in preparation programs for educational administrators” (p. 2, 1973). The 
authors also examined how training in these methods could help school administrators address the 
rising dual demands of accountability and instructional improvement and provided overview 
summaries on the content of multiple university programs (Bruno & Fox, 1973). Their report 
provided useful evidence in their time, but the findings of Bruno and Fox (1973) have not been 
updated over the 40 years following their efforts (Bowers, 2017). Through the present report we 
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provide an overview of the emerging literature around this topic and work to illuminate new and 
strengthened evidence as well as learning to further our on-going efforts to enhance leadership and 
school performance with data and new ways of using data. 


Emergence of Data Analytics in Education Leadership 

The use of data for decision-making in education is “neither a new topic nor an unknown practice” 
with “growing awareness dating back to the 1990s [when] school principals, teachers, parents, 
stakeholders and policymakers started looking at quantitative data as an indispensable source for 
making decisions, formulating diagnoses about strengthens and weaknesses of institutions and 
assessing the effects of initiatives and polices” (Agasisti & Bowers, 2017). Over the past several 
decades, quantitative research methods courses in colleges of education have thus become essential 
in preparing school district administrators and leaders to effectively manage school systems 
(Agasisti & Bowers, 2017; Bowers, 2017; Bruno & Fox, 1973; Kowalski, McCord, Peterson, 
Young, & Ellerson, 2011). Brunto and Fox (1973) attribute earlier growing demand for 
quantitative methods to the accountability movement and the development of new tools and 
techniques of their time that allowed for “the quantification of information for decision making,” 
which is otherwise known as “quantitative analysis” (p. 2). Since then, there has been a long 
tradition of education leadership graduate programs that include at least one quantitative research 
course alongside other courses deemed necessary for effective leadership in education (Anderson 
& Reynolds, 2015; Bowers, 2017; Hess & Kelly, 2005; Militello, Gajda, & Bowers, 2009; 
Thornton & Perreault, 2002). In addition, doctoral PhD or Ed.D. programs in education leadership 
have been on the rise with increasing numbers of programs and graduates of these programs in the 
United States (B. D. Baker, Orr, & Young, 2007; Goldring & Schuermann, 2009; Hackmann & 
McCarthy, 2011). With this rise, quantitative methods courses have continued to be offered and 
many times taken for granted in university degree programs (Bowers, 2017). Nevertheless, in the 
last 50 years, and especially so in the last 15 years, the field has seen an increasing trend of growing 
high-quality education research across domains (Bowers, 2017; Wang & Bowers, 2016; Wang, 
Bowers, & Fikis, 2016) along with a growing diversity of quantitative methods (Bowers, 2017; 
Goff & Finch, 2015; Hallinger & Heck, 2011), and the emergence of learning analytics (R. S. 
Baker & Inventado, 2014; Bienkowski, Feng, & Means, 2012; Piety & Pea, 2018). With the 
plethora of research published and increasing sophistication of research methods, a central purpose 
of leadership training has been “to train future school system leaders to become consumers of this 
work and apply critical thinking and evaluation of analytics to their decisions in their schools on a 
daily basis” (Bowers, 2017, p. 78). 


Bowers (2017) examines the current state of quantitative research methods courses in education 
leadership and offers useful suggestions for curricula and instructional techniques that can help 
prepare graduates of education leadership programs use data and research regularly to foster 
improvement in instruction in schools, while strengthening collaboration between universities and 
districts. In addition to teaching basic statistics and empirical reasoning, he states that “quantitative 
methods courses provide an opportunity to build the capacity of school leaders as practitioner- 
scholars in assessment literacy, data literacy, and how to facilitate and lead building professional 
capacity through evidence-based improvement cycles” (2017, p. 74). Moreover, he discusses 
findings from the literature review on the different data and analysis needs of teachers, principals, 
and superintendents and asserts the need to provide differentiated recommendations for 
encouraging data use that reflects differing perspectives. Yet, ever more data continues to become 
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available in schools, while data strategists in districts and states must focus on compliance rather 
than analysis, understanding, application, and working with school and district leaders to 
communicate to stakeholders. 


In the last decade, we have witnessed strong growth in industry, technology and the sciences in 
big data, data analytics, and the emerging domain of data science (Cope & Kalantzis, 2016; 
Donoho, 2015; Gandomi & Haider, 2015; Schutt & O'Neil, 2013). As noted recently by Krumm 
et al. (2018) “The daily activities of schools and universities— from taking attendance to assessing 
students— can leave a trail of data that, under the right conditions, can be used to explore teaching 
and learning like never before.” (p.1). Nevertheless, compliance focus around data persists, while 
schools and districts remain limited in their capacities to meet the increasing demands to leverage 
their now abundant data to drive decisions. One reason for this is that nationally, district data 
strategists have had few to no options to acquire training and certification around data use, data 
integration and management, “e-administration”, data mining, coding, statistics, and most 
importantly, how to collaborate and work with management in a way that would help leaders build 
capacity within organizations for evidence-based improvement cycles that involve conversations 
using data and evidence (Agasisti & Bowers, 2017; Bowers, 2017; Bowers, Krumm, Feng, & 
Podkul, 2016). To meet this need, “Organization-Level Data Analytics” or “Education Leadership 
Data Analytics” (ELDA) has been proposed (Agasisti & Bowers, 2017; Bowers, 2017) as a new 
domain that would help build the capacity of school and district data strategists as well as school 
administrators, educational quantitative analysts, research specialists, and data scientists to 
effectively understand and use data to inform decisions and strengthen collaboration to improve 
school systems across the United States as well as internationally. 


Professional and Academic Rationale for ELDA 
Education Leadership Data Analytics (ELDA) is the emerging domain of research, practice, and 


policy in education that focuses on bringing together three broad domains at the intersection of 1) 
Education Leadership, 2) Evidence Based Improvement Cycles, and 3) Data Science (see Figure 


1). 


Education Leadership Data Analytics (ELDA) Definition: 


Education Leadership Data Analytics (ELDA) practitioners work collaboratively with schooling 
system leaders and teachers to analyze, pattern, and visualize previously unknown patterns and 
information from the vast sets of data collected by schooling organizations, and then integrate 
findings in easy to understand language and digital tools into collaborative and community- 
building evidence-based improvement cycles with stakeholders. 
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Figure 1: Education Leadership Data Analytics (ELDA) Venn Diagram 


Current research in data-driven decision-making and leadership in education has called attention 
to a significant need for education leaders to develop capacities in these three main areas. On June 
8, 2018, the first Education Leadership Data Analytics Summit was held at Teachers College, 
Columbia University. The Summit was designed to bring together education and research leaders 
and innovators working at the intersections of individual overlaps of two or three of these three 
circles in the Venn diagram. The ELDA Summit brought participants together across a wide 
variety of fields in education. ELDA practitioners work collaboratively with or within schools and 
districts, know how to facilitate and lead capacity-building conversations with educators using 
evidence and data to build trust and collaboration, and work with data through code, analytics, data 
mining and machine learning, and statistics to visualize and pattern data in new and informative 
ways. 


Increased capacity building and training in education preparation programs in data literacy pertain 
to the training of educators in issues of assessment, measurement, and data use in practice 
(Mandinach et al., 2015; Mandinach & Gummer, 2013). From the perspective of leadership and 
evidence-based improvement cycles, the research literature has recently started to shift away from 
emphasizing data-driven decision making (Bambrick-Santoyo, 2010), where leaders infuse data 
into their decision-making process, towards more holistic approaches of evidenced-based 
improvement cycles where school leaders are encouraged to use evidence to develop capacity of 
teachers within their organization and build a trusting culture in which teachers collaborate and 
use evidence to help each other improve their instruction (Boudett, City, & Murnane, 2013; 
Bowers, 2017; Datnow & Hubbard, 2015; Hoogland et al., 2016; Marsh, 2012; Marsh & Farrell, 
2015; Schildkamp et al., 2017; Supovitz & Morrison, 2015). These processes have worked to help 
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organizations move from traditional high inference, low evidence conversations to high evidence, 
low inference conversations that strengthen capacity for instructional improvement (Agasisti & 
Bowers, 2017; Bowers et al., 2014). 


Recently, we seen the rise of big data analytics to potentially inform education and education 
decision making (Agasisti & Bowers, 2017; Cope & Kalantzis, 2016; Halverson et al., 2015; 
Krumm et al., 2018; Piety et al., 2014; Singh et al., 2017). The application of data science to 
organizational improvement in education (Bowers et al., 2016; Krumm et al., 2018; Piety et al., 
2014; Piety & Pea, 2018) refers to the work of analysts who “translate data into knowledge and 
action through data mining and visualization, but also through interfacing with organizational 
leaders and stakeholders to inform evidence-based decisions” (Agasisti & Bowers, 2017) (p.187). 
Through the use of data science, education data mining (EDM), learning analytics (LA), and visual 
data analytics, the goal of this work with education data is to “make visible data that have 
heretofore gone unseen, unnoticed, and therefore unactionable” (Bienkowski et al., 2012) (p.ix). 


Agasisti and Bowers (2017) identify and suggest concrete steps to further the development of data 
analytics in education that move beyond data use to the diffusion of analytics in supporting 
processes of educational decision-making that require strengthened collaboration among actors 
who make decisions and those who work with data in school systems. In their view: 


Despite the rapid growth of attention towards the role of data and quantitative 
information for exploring and analyzing educational patterns and results, there is 
still a relevant separation between decision makers (principals and middle 
managers at the institution level, politicians at the governmental level) and data 
analysts and researchers (2017) (p.190). 


In many ways, research to date indicates that school decision makers are aware of the importance 
and potential of data, yet do not consider their technical expertise as adequate to use data, while 
data analysts are satisfied with their technical expertise and work with data without connecting to 
the practical aspects of managing and improving schools (Agasisti & Bowers, 2017; Krumm et al., 
2018). To bridge this disconnect, researchers in this domain have proposed a new role, namely that 
of the “education data scientist” (Agasisti & Bowers, 2017; Bowers et al., 2016; Cope & Kalantzis, 
2016; Krumm et al., 2018; Piety et al., 2014; Piety & Pea, 2018). 


Potential Challenges 

Integrating data analytics in education leadership, with its incredible potential, will also come with 
a set of challenges. Researchers have identified and discussed many barriers and impediments to 
the use of analytics in education (Agasisti & Bowers, 2017; Cope & Kalantzis, 2016; Krumm et 
al., 2018; Piety et al., 2014; Piety & Pea, 2018). These challenges pertain to the area of ethics, 
complexity, cost, and school capacity. The problem around ethics is rooted in privacy issues and 
the potentially intrusive use of data and access to personal information (Agasisti & Bowers, 2017; 
Cope & Kalantzis, 2016; Ifenthaler & Tracey, 2016; Spector, 2016). The second challenge around 
complexity is more technical in nature and pertains to the complexity of data and data integration 
and the need for adequate systems of data storing and organization as well as robust data that is 
accurate and reliable (Agasisti & Bowers, 2017; Cope & Kalantzis, 2016; Piety et al., 2014). Bruno 
and Fox (1973) also discuss problems arising from the nature of quantitative analysis for which its 
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“complexity leads to confusion” (p. 19). Third, introducing and integrating data analytics for 
decision making in schools is a costly investment that requires building new infrastructure and 
sufficient resources both human and material (Agasisti & Bowers, 2017; Piety et al., 2014; 
Wayman, 2005; Wayman, Cho, & Johnston, 2007; Williamson, 2016). Finally, the challenge 
around school capacity pertains to “whether the various institutions that are interested in 
developing sophisticated analyses actually have the necessary competences, in terms of technical 
(analytical) skills of the personnel or, at least, they create conditions for developing them” 
(Agasisti & Bowers, 2017) (p.203). To address these potential barriers and impediments, Agasisti 
and Bowers (2017) suggest the use of ethical guidelines, processes and systems to manage 
increasingly complex data and ensure accuracy, utilization of open access code, and the 
development and provision of training. 


Four domains for Training in ELDA: 

As mentioned above, a central concern of the ELDA field is developing training and certification 
programs in Education Leadership Data Analytics (ELDA) that would build the capacities of 
various educational actors to effectively use data to inform decisions and strengthen collaboration 
to improve school systems across the United States as well as internationally. Here we would like 
to identify who these actors are and how this work in ELDA seeks to address their capacity needs 
and points of systemic collaboration. Bowers (2017) builds from the work of Bruno and Fox (1973) 
and identifies at least four different roles in the domain of education leadership and data analytics 
in schools. These differentiated roles— the first three discussed by Bruno and Fox (1973) and the 
fourth identified by Bowers (2017)—point to the need for current graduate programs to go beyond 
training for general school administration to build capacity for effective data-driven decision- 
making in schools. These roles are further described below and in Figure 2. 
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Primary Quantitative Program 
Focus on evidence-based 


improvement cycles Practicing 
* Quantitative research methods Administrator 
* Journal clubs 

University-district partnerships 


Evaluation & Analysis Focus Educational 
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Psychometrics & testing 
Data management & ethics 
Survey & program evaluation 
Cost-benefit analysis 
Accounting & budgeting 


Analyst 


Research 
Specialist 


Organization-Level Data Analytics 

* Education data mining 
Learning analytics 
Programming 


Technology & instruction Data Scientist 
Design-based research 


Figure 2. A model for quantitative research methods training in education leadership graduate 
programs. (from Bowers 2017) 


The Practicing Administrator 

The “Practicing Administrator,” are current or aspiring superintendents and school administrators 
for whom traditional quantitative methods courses have been focused on training (Bowers, 2017; 
Bruno & Fox, 1973). They are the decision-makers who will manage school systems and oversee 
efforts to improve organizational performance. With respect to efforts in ELDA, this person may 
already understand the importance of data, even using descriptive data to monitor their school 
performance, and would need to be trained in how to use data and analytics to make decisions, 
address problems, and inform evidence-based improvement cycles. This work should be focused 
on helping the practicing administrator know what types of questions can be asked and adequately 
answered using advanced data analytics, but also how to integrate those findings into evidence- 
based improvement cycles. These evidence-based improvement cycles should focus almost 
exclusively on the practicing administrator having the capacity and skills to facilitate positive 
collaborative conversations between and among teachers in their organizations in which the 
evidence is used to build trust, collaboration, and capacity through teachers coming together in 
evidence-based improvement cycles to improve their specific instruction around issues they 
identify with their students. 
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The Educational Quantitative Analyst 

The “Educational Quantitative Analyst” is the person who is responsible for using and managing 
the day-to-day operational management data for the organization (Bowers, 2017; Bruno & Fox, 
1973). These include assessment and test reporting to policy makers, enrollment trends and 
projections, addressing issues of accounting and cost—benefit analysis, searching for efficiencies 
in the system, analyzing data for bus routes and curriculum scheduling, and collecting data for 
personnel evaluations. This person focuses on “efficient service management analytics” and may 
benefit from learning more about how their work fits with the work of other actors in the system 
and how to strengthen collaboration through data analytics. 


The Research Specialist 

The “Research Specialist” conducts research on and with the organization, which includes 
analyzing effectiveness, efficiency, and instructional improvement (Bowers, 2016; Bruno & Fox, 
1973). A contemporary example of this type of position could be researchers in organizations such 
as the University of Chicago Consortium on Chicago School Research (CCSR), a university- 
district partnership in which core research questions that are of interest to the Chicago Public 
Schools are addressed by researchers at CCSR through a mutually beneficial collaborative 
cooperation agreement (Bowers, 2017; Roderick, Easton, & Sebring, 2009). This person “focuses 
on assessment construction and validation, surveys and program evaluation with an eye towards 
psychometrics, testing and inferential statistics” (Agasisti & Bowers, 2017) (p.204). Similar to 
the educational quantitative analyst, this person may benefit from seeing how their work aligns 
with the work of other actors in the system and how they may strengthen their collaboration 
through the use of analytics. 


The Education Data Scientist 

There is a fourth type of quantitative education leadership practitioner-scholar position that has 
emerged over the last 40 years, the Education Data Scientist. Bowers (2017) uses the following 
definition of a data scientist from Schutt and O’Neil (2013): 


A data scientist is someone who knows how to extract meaning from and interpret 
data, which requires both tools and methods from statistics and machine learning, 
as well as being human . . . Once she gets the data into shape, a crucial part is 
exploratory data analysis, which combines visualization and data sense . . . She’ ll 
find patterns, build models, and algorithms . . . She may design experiments and is 
a critical part of data-driven decision making. She’ll communicate with team 
members, engineers, and leadership in clear language and with data visualizations 
so that even if her colleagues are not immersed in the data themselves, they will 
understand the implications. (Schutt & O’Neil, 2013, p. 16) 


This definition fits with the description from Agasisti and Bowers (2017) and echoes the 
recommendations of both Cope and Kalantzis (2016) and Piety et al. (2014) to understand the 
educational data scientist as one who “owns the technical skills to collect, analyze and use 
quantitative data, and at the same time the managerial and communication skills to interact with 
decision makers and managers at the school level to individuate good ways of using the 
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information in the practical way of improving practices and initiatives” (Agasisti & Bowers, 2017) 
p.190. 


Summary: 

Together, these four roles of the practicing administrator, education quantitative analyst, research 
specialist, and education data scientist, make up four different facets of the work of Education 
Leadership Data Analytics (ELDA) practitioners. Current ELDA practitioners are the people in 
districts, states, and nations who help to lead their organizations using data and analytics. Put 
simply, these are the people in each schooling system who everyone knows as the person who 
“does data”. As noted above, doing this work is a complex task of human interaction and people 
skills, leadership skills, as well as psychometrics, statistics, evaluation, machine learning, coding, 
database management, data visualization, and with all education leadership roles, politics. A 
daunting job to say the least. In sum, while there is an emerging set of research on education data 
science as well as the ELDA intersection of education leadership, evidence-based improvement 
cycles, and data science/data analytics, there appears to be an opportunity to provide professional 
development and capacity training in this field, as the research currently indicates that the field is 
underserved. 


An Example: The Harvard Strategic Data Project 

While the research to date discussed above has noted the dearth of training, networking, and 
professional development activities for these types of education professionals, the Harvard 
University Strategic Data Project (SDP), is one of the few exceptions and an excellent example of 
an organized intention to build networking and training capacity in the field. Originally funded 
through a grant from the Bill and Melinda Gates Foundation, Harvard SDP annually recruits 
district data strategists who either join as fellows in the program or are already currently working 
in a state or local education agency (Center for Education Policy Research, n.d.; Hallgren, Pickens 
Jewell, Kamler, Hartog, & Gothro, 2013). These participants then meet regularly as a cohort during 
the year long program, are paired with national faculty (not necessarily at Harvard), and work on 
a data analytics and evidence-use project in their schooling organization that is designed to address 
a specific data issue or problem. Along with providing data analysis tools, code, and protocols for 
using data to inform decisions in schooling organizations, Harvard SDP is one of the few national 
and global opportunities for data strategists to build their skills in their career, as well as network 
and share practices, code, analytics, and professional development with other likeminded data 
strategists. 


Structure of this Report: 

We turn next in this report to an overview of the ELDA Summit 2018 event. We then discuss the 
outcomes from the summit, provide a blueprint of possible skills and competencies for training 
ELDA practitioners, and then conclude with recommendations for the field. 
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Education Leadership Data Analytics Summit 2018 
ELDA Summit Overview and Description of the Speakers, Topics, and Event Structure: 


On June 7 and 8 of 2018, 130 researchers and practitioners gathered at Teachers College, Columbia 
University in New York City to network and collaborate around the topic of Education Leadership 
Data Analytics (ELDA) at the ELDA Summit 2018. As described above, ELDA is an emerging 
domain that brings together Education Leadership, Evidence-Based Improvement Cycles, and 
Data Science to help schools, districts, states, and nations put the data they already collect to better 
use to inform decisions and capacity building of educators through applying big data and data 
science techniques to education data. The event started with a networking and poster session the 
evening of June 7 with 25 posters presented on a range of research and practice topics. Then on 
June 8, participants gathered first in the morning sessions for a keynote address by Andrew Krumm 
from Digital Promise, discussing his work in personalized learning and evidence use. This was 
followed by an engaging panel discussion that included representatives from Harvard, IBM 
Watson, University of Delaware, and the Learning Analytics, Education Quality Assurance, and 
Education Leadership Data Analytics groups at Teachers College, Columbia University. In the 
afternoon, participants gathered in the Smith Learning Theater at Teachers College, Columbia 
University, to network and collaborate in interactive teams where they heard from representatives 
from MIT, Digital Promise, Rochester City School District, and Fresno Unified School District on 
current data practices, data visualization, and open and FAIR data practices. 


The event brought together researchers, practitioners, funders, and policymakers with nationally- 
recognized researchers to discuss the challenges and opportunities in this emerging field. 


The Twitter event hashtag was #ELDASummit18 
https://twitter.com/search?f=tweets &vertical=default&q=%23ELDAsummit18&src=typd 


The agenda for the two-day event was: 
Thursday, June 7: Horace Mann 138 & 140 


5:00pm - 7:00pm Pre-summit Networking Reception and Poster Presentations 


Friday morning, June 8: Milbank Chapel 
8:30am - 9:00am Registration 


9:00am - 9:15am Welcome Remarks 
Mark Anthony Gooden, Christian A. Johnson Endeavor Professor in Education Leadership, 
Teachers College, Columbia University 


9:15am-9:30am Introduction to Education Leadership Data Analytics 
Alex J. Bowers, Associate Professor of Education Leadership, Teachers College, Columbia 
University 


9:30am - 10:15am 


Keynote: Learning Analytics Goes to School: A Collaborative Approach to Improving Education 
Andrew Krumm, Director of Learning Analytics Research at Digital Promise 
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10:30am-12:00pm 
Panel: The Multiple Facets of Education Leadership, Evidence Use, and Data Analytics 


Madhabi Chatterji, Professor of Measurement, Evaluation and Education, Teachers 
College, Columbia University 

Elizabeth Farley- Ripple, Associate Professor of education and public policy, 
University of Delaware 

Gary Natriello, Ruth L. Gottesman Professor in Education Research & Professor of 
Sociology and Education, Department of Human Development at Teachers College, 
Columbia University 

Miriam Greenberg, Director of Education and Communications, Center for Education 
Policy Research, Strategic Data Project, Harvard University 

Alex Kaplan, Global Leader, Large Deals, IBM Watson Education 

Kenny Graves, Doctoral Candidate in Education Leadership, Teachers College, 
Columbia University 


Friday noon, June 8: Grace Dodge 177 & 179 
12:00pm - 1:30pm ~—— Lunch 


Friday afternoon, June 8: Smith Learning Theater 
1:30pm - 4:45pm Education Leadership Data Analytics Collaborative Workshop 


Alex J. Bowers, Associate Professor of Education Leadership, Teachers College, 
Columbia University 

Kenny Graves, Doctoral Candidate in Education Leadership, Teachers College, 
Columbia University 

Quick-Talk visualization and collaborative exemplars 

Andrew Krumm, Director of Learning Analytics Research at Digital Promise 

Amy Nurnberger, Program Head, Data Management Services, Massachusetts Institute 
of Technology & Adjunct Assistant Professor, Learning Analytics Program, Teachers 
College, Columbia University 

Jing Che, Senior Research Analyst at Rochester City School District (NY) 

David Jansen, Executive Director, Data Science and Software Systems at Fresno 
Unified School District 


4:45pm — 5:00pm Closing 
Alex J. Bowers, Associate Professor of Education Leadership, Teachers College, Columbia 


University 
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Poster of Presenters for the ELDA Summit 2018: 


EDUCATION 
LEADERSHIP DATA 
ANALYTICS SUMMIT 
2018 


Dr. Mark Anthony Gooden Dr. Alex J. Bowers Dr. Andrew Krumm Dé Macha atta 
@gooden_mark @Alex_J_Bowers @aekrumm ‘ a 
Teachers College, Teachers College, Digital Promise Teachers College, Columbia 

Columbia University Columbia University University 


Dr. Elizabeth Farley-Ripple 
@FarleyRipple 


Miriam Greenberg 
@mirigreenberg 


Dr. Gary Natriello Alex Kaplan 


University of Delaware Teachers College, Harvard University IBM Watson Education 
Columbia University 


; ba ie 
Kenneth E. Graves Amy Nurnberger Dr. Jing Che David Jansen 
@kegice @ANurnberger @genieche @davidejansen 
Teachers College, Columbia MIT Rochester City School Fresno Unified School 
University District District 
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Overall Organization: 

The meeting was organized into three parts. First, on the Thursday night, participants were invited 
to present posters on their work in the ELDA space. For the poster session, we had 25 posters 
presented with over 60 attendees. For the Friday morning introduction, keynote, and panel 
discussion, and lunch, we had 120 attendees. The Friday afternoon interactive event in the Smith 
Learning Theater had 100 attendees (maximum capacity). 


Thursday night June 7 poster session 
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Friday morning opening, introduction, keynote, and panel discussion 
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Friday afternoon: Smith Learning Theater interactive event 
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Summit Meeting Intention and Goals: 

The workshop/conference kicked off with a pre-reception and poster session for attendees on the 
night of Thursday June 7 to welcome folks and give all attendees an opportunity to showcase their 
work and network with other attendees. Then on June 8, we invited speakers from across the 
country for talks and panels in the morning and afternoon on what ELDA is, and how data 
analytics, data science, and big data practices that are drawn from "big data for the social good" 
domains can help inform decision making, leadership, and school evidence-based improvement 
cycles. In the afternoon, we gathered 100 attendees together for an interactive collaborative session 
in the Smith Learning Theater, and had them work in 20 teams of 5 to uncover the big issues in 
the ELDA domain, both in doing the work and in building a pipeline of people to inform the work, 
with the outcome as research questions, areas of focus, and networking across the teams to build 
capacity in the domain. The intention of the summit was to have the attendees build their capacity 
and ideas around ELDA through the poster session and morning keynote and panel discussions, 
and then mine that information in the collaborative teams in the afternoon in an interactive session 
where we would network the participants together, and then build the ideas together for what 
ELDA is and where the field could go. 


Topics for the morning session included: 


e Mark Gooden, Teachers College, Columbia University — Opening welcome as the Director of 
the Education Leadership Program at Teachers College, Columbia University, with a focus 
on how data analytics and data use can help focus schooling organizations on addressing the 
needs of all children. https://www.tc.columbia.edu/organization-and-leadership/education- 


leadership/ 


e Alex Bowers, Teachers College, Columbia University — Opening introduction to ELDA, 
outlining the Venn diagram of Education Leadership, Evidence-based Improvement Cycles, 
and Data Analytics/Data Science (Agasisti & Bowers, 2017; Bowers, 2017; Bowers et al., 
2016). 


e Andrew Krumm, Digital Promise — Keynote address discussing the application of ELDA to 
personalized learning through discussing his new co-authored book, Learning Analytics Goes 
to School: A Collaborative Approach to Improving Education (Krumm et al., 2018), which 
details the work through an NSF grant with charter schools in Silicon Valley in California 
and personalized learning. 


e Madhabi Chatterji, Teachers College, Columbia University — Panel discussion on her work 
on quality assurance in education and her work on the Assessment and Evaluation Research 
Initiative at Teachers College, Columbia University. https://www.tc.columbia.edu/aeri/ 


e Elizabeth Farley- Ripple, University of Delaware — Panel discussion on her work as co- 
principal investigator for the Center for Research Use in Education. 
http://www.research4schools.org/ 
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e Gary Natriello, Teachers College, Columbia University — Panel discussion on his work 
directing the Learning Analytics program at Teachers College, Columbia University. 
https://www.tc.columbia.edu/human-development/learning-analytics/ 


e Miriam Greenberg, Harvard University — Panel discussion on her work helping to direct the 
Strategic Data Project at the Center for Education Policy Research. 


https://sdp.cepr.harvard.edu/ 


e Alex Kaplan - IBM Watson Education — Panel discussion on his work on the role IBM and 
AI plays in education data science. https://www.ibm.com/watson/education 


e Kenny Graves, Teachers College, Columbia University — Panel discussion on his work on the 
intersection of school technology, school leadership, and social justice (Graves & DeLyser, 
2017). 


Afternoon session in the Teachers College, Columbia University Smith Learning Theater 


( sn For the afternoon Learning Theater session, we built upon our experience 
from previous research and examples using design-based action-oriented 
|Z interactive events (Finkel, Graves, & DeLyser, 2017), in which first 


Welcome 


participants were placed into teams of 5 based on their responses to a pre- 
event registration survey that asked open-ended response items as to their 
interest in ELDA and their expectations of the event. We had an icebreaker 
followed by four rounds of quick talks from work “from the field” in which 
a quick talk speaker gave a 10 minute presentation, and then teams were 
asked to digest that information together by creating post-it notes that 
broke down the issues for research and practice around ELDA from what 
they had just heard. 


| 
Education Leadership | 
Data Analytics 


Workshop 
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As an overview of the afternoon Learning Theater 
session, the following link leads to a two-minute video 
overview of the event, providing an opportunity to see the 
collaborative nature of the event. 


http://go.tc.edu/DataAnalyticsSpotlight 


h % ~ 
Andrew Krumm, Digital Promise — Providing additional “deep dive” information on how the work 
was done for his new book in collaboration with personalized learning charter schools, in which 
he detailed cluster analysis heatmap data dashboard creation and use by school leaders to make 
decisions (Krumm et al., 2018). 


“= 
Amy Nurnberger, Massachusetts Institute of Technology — Provided a deep dive on FAIR data 
standards in education research, that data are Findable, Accessible, Interoperable, and Reuseable. 
This is the opposite of FOUL data that are Frustrating, Obfuscated, Unmanaged, and Lost (Austin 
et al., 2017). 
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Jing Che, Rochester City School District (NY) — Provided a deep dive on using ELDA in practice 
in Rochester to inform leadership decisions around student at-risk indicators (Che, 2016). 


David Jansen, Fresno Unified School District — Provided a deep dive on the student interaction 
app and system in Fresno that has allowed them to make more informed decisions on student 
attendance (Garcia Mathewson, 2017). 


Once participants had generated multiple 
post-it notes with ideas for research and 
practice in ELDA based on the content of 
each quick-talk, teams then clustered Post-it 
notes together, labeled clusters, and created 
topics, questions, and issues in ELDA that 
were most relevant to them. Teams then 
placed these topics and questions in two 
dimensions, priority and possibility, rating 
each one from | to 5, and then graphing 
them, to understand which issues had the 
highest possibility and priority. 
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During this activity, camera crews roamed the room, 
and displayed the work of individual teams on the 
big screens as well as multiple large LCD monitors 
throughout the Learning Theater. In this way, 
individual teams could see what the other teams were 
working on, and some information from other teams 
could make its way across teams through the video 
displays. 


so that all teams could see the 20 top priority versus 
possibility choices. These were then put into a Qualtrics 
survey in real time, and a Qualtrics online survey was 
posted for participants to fill out. All participants then 
completed the survey where they rated each of the 20 
ideas/issues/concerns around ELDA on a scale from 1- 
5 on priority versus possibility. The overall average ratings were displayed in two dimensions on 
the screens. Teams were given time to discuss what they saw. Then, each table had a location 
tracking device, called a Quppa device, one for each participant. Participants picked these up, and 
walked around the room, doing a gallery walk and dropping their device on a table for a team that 
they wanted to hear more from. The location of each device was displayed on a map of the Learning 
Theater so that participants could see where the devices piled up. We then did a selective share- 
out from teams and ended the day with a brief summary and thank-you to the participants, staff, 
and volunteers. 


From this event, participants filled out three different surveys as a means to collect the collective 
knowledge of the participants in the summit. Participants filled out a registration survey, the 
Learning Theater real-time possibility versus priority survey, and a post-event survey. 
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Data Analysis of Participant Summary Survey Responses: 
Registration Summary Survey Data Analysis: 


Prior to the meeting, attendees filled out a brief registration survey, identifying their job role and 
industry sector, the coding tools that they use to do their work, as well as writing a few words on 
open response items about what they thought about ELDA as a field, and their goals for attending 
the meeting. We analyzed this data in two ways. First, we created counts of registrants and provide 
the bar graphs below of the percentage of registrants from each job role, field, as well as preferred 
data analysis tools. Second, we generated a wordcloud to visualize the most frequent words used 
in the open-ended response items. 


In Figures 3 and 4, the percentages of registrants show that the majority of participants were from 
higher education and were researchers and analysts, mostly studying K-12 and higher education 
issues. While there were some practitioners, such as teachers and school leaders, practitioners were 
a group that had low representation at the summit. This issue was noted by a few participants in 
the post-event survey as well. Interpretation of outcomes from the summit should take this issue 
into account. 


Job Role of Registrants 


Researcher ETT 
University Faculty DA 
Data Scientist 
Analyst 
Consultant EE 
Graduate Student EE 
K-12 School Administrator [i 
Higher Ed Admin Sl 
Funder ml 


Teacher 


re 
K-12 District Leader @ 
Educational Software Developer @ 
Education eCommerce & 
Non-profit Leader ff 


0 10 20 30 40 50 60 70 80 90 100 
Percent of Registrants 


Figure 3: Job role of registrants. In Figure 3, of the 130 registrants, the largest percentage 
responded that their primary job role was as a researcher or university faculty. This accounted for 
over 2/3" of the registrants. Less than 5% of registrants for the summit were teachers or district 
leaders. 
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Figure 4: Industry sector of registrants. Figure 4 shows that of the 130 registrants, almost half 
were in higher education, while over 40% were K-12 education focused. 


In Figure 5, we asked what statistical or data analytics software registrants were most proficient 
using. The results help to understand what people interested in ELDA know as far as software, 
with the top programs being the open source R statistical software, SPSS, Excel, STATA, and 
Python. To our knowledge, this is one of the first surveys of software usage in the ELDA or 
education data sciences field. 


Which software packages do you feel proficient in for statistics, data science, 
and data mining? 
—R 
SPSS 
Stata 
Excel 
Python 
Mplus 
SQL 
Tableau 
SAS 
Power BI 
Rapidminer 
UCINET 
Matlab 
Qualitative Only 


oO 
oO 
N 
oO 
Ww 
oO 
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Figure 5: Software packages used by registrants. Of the 130 registrants, the most popular software 
package used was the open source R statistical software package, followed by SPSS, STATA, and 
then Excel and Python. Additionally, more than 10% of registrants noted that qualitative research 
methods were their primary proficiency. 
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Figure 6: Wordcloud of Registrant Responses to Open-Ended Items. Registrants were asked two 
questions: 1) Please describe your prior experience as well as role and/or involvement with 
education leadership and data analytics and 2) Please let us know how our convening could be 
helpful to you in your work. Figure is a wordcloud in which for the top 100 frequently occurring 
words, a larger font size indicates greater frequency. 


Figure 6 provides a descriptive summary of the general interests of the registrants. The most 
frequent words were the title of the ELDA summit itself, including data, education, leadership and 
analytics. Additionally, highly frequently mentioned terms also included learning, research, 
school, as well as a many related words to these topics. Data was by far the most frequently 
included word. 
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Learning Theater Survey Data Analysis: 
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[Poor quality data will make everything else invalid. 


|Co-creation: Just because it's difficult doesn’t mean you shouldn't do it. 
— 


> 
3.000 Proper training in research design has the ability to produce real change that can drive continuous improvement. 


We are so enamored with the possibilities of big data applications that we've overlooked fundamental questions about the purpose, ethics, and ultility for human-centered concerns. 
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Figure 7: Priority versus Possibility plot from Learning Theater event 


As noted above, as part of the Smith Learning Theater interactive afternoon event, each participant 
rated the ideas/issues/topics that the 20 teams generated from the event from 1-5 (5 being highest) 
in both priority and possibility dimensions. Figure 7 is the figure that was displayed to the 
participants during the event. The axis in both the x and y dimensions is shortened to display the 
topics in a way that makes them readable, as all of the issues cluster into a fairly tight cluster in 
the upper right of the overall plot, as the cluster is above 2.5 in each dimension. This figure 
represents the summary averages across all 100 participants for the 20 top identified topics in 
ELDA. 


The goal of this process was to visualize the distribution of what are the highest priority and highest 
possibility topics, issues, and ideas that surfaced for the participants for the field of ELDA, and 
capture this information. The top issues in the upper right corner with the highest priority and 
possibility were: 


— 


Our priority is human capacity for ELDA: to do it well and for stakeholders to use it well. 

2. Data quality and management matter most because without those, nothing good can come from 
learning analytics. 

3. A healthy culture of inquiry guides why we ask the questions we do and how we pursue 

meaningful and actionable information to make impactful decisions. 
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1. Develop and foster effective and ethical partnerships between researchers and practitioners in 


order to use data to drive quality education 
2. Clear directions for HOW and WHY to use data properly. 
3. Poor quality data will make everything else invalid. 


Additionally, the below table provides the average priority and possibility data for each topic 


ordered by highest to lowest average priority rating. 


Description Priority | Possibility 
Develop and foster effective and ethical partnerships between researchers and 

practitioners in order to use data to drive quality education 4.09 3.35 
Clear directions for HOW and WHY to use data properly. 4.02 3.62 
Poor quality data will make everything else invalid. 4.00 3.15 
Our priority is human capacity for ELDA: to do it well and for stakeholders to use it 

well. 3.98 3.4 
A healthy culture of inquiry guides why we ask the questions we do and how we pursue 

meaningful and actionable information to make impactful decisions. 3.98 3.27 
How do you get students and teachers meaningfully involved in developing, 

implementing, improving learning analytic systems? 3.87 3.2 
Data quality and management matter most because without those, nothing good can 

come from learning analytics. 3.87 3.44 
Without the skills to collect, analyze, interpret, and communicate data, educational 

improvement is random or uncertain. 3.82 3.22 
Ensuring that the usage of our data analytics is equitable for everyone 3.76 2.62 
Buy-in and commitment to establishing or changing Data culture is most important and 

most difficult. 3.71 3.2 
Training empowers execution to bring about desired outcomes! 3.69 3.45 
We are so enamored with the possibilities of big data applications that we've overlooked 

fundamental questions about the purpose, ethics, and ultility for human-centered 

concerns. 3.69 2.82 
Impact is the most critical element of ELDA for the sustainability of school reform. 3.62 2.82 
Sharing (fair) data/ research/thought process/ results is caring for your community of 

learners. 3.60 3.27 
Pictures and Tables without Practices for Teaching are Pointless and Tiresome. 3.56 3.47 
Co-creation: Just because it's difficult doesn't mean you shouldn't do it. 3.55 3.05 
Access to data is the foundation for data analytics in education leadership. 3.49 3.38 
investigation into development of applications requires effective communication to 

create buy in and build capacity 3.44 3.22 
Proper training in research design has the ability to produce real change that can drive 

continuous improvement. 3.44 3.04 


Overall, attendees for the Learning Theater event appear to want ELDA as a field to focus on 
building capacity in the field, focusing on data quality and data management, and building a culture 
of inquiry in schools, with additional areas of focus that include ethical and useful researcher 
practitioner partnerships, clear directions on data use, and a focus on ensuring data quality. These 
are central issues to consider in any professional development offerings to the ELDA field. 
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Post-Event Summary Survey Data Analysis: 


A few days after the conclusion of the event, participants were sent a Qualtrics post-event survey 
by email. There were 47 respondents from 130 participants for a 36% response rate. Across the 
survey, participants responded that they were very satisfied by the overall event, and found others 
to network with in the field throughout the event. 


In Figure 8 below, participants rated the central issues for the ELDA field from not important to 


99 66 


very important. The top five most important areas were “hearing from practitioners”, “capacity 


9 oe 


building and training for practitioners”, “evidence-based improvement cycles/data use”, “data 
science/data analytics”, and “hearing about research practice partnerships”. 


Average Importance of Domain within ELDA 
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Figure 8: Average post-event survey responses (0: not important to 3: very important) for each 
domain discussed during the ELDA Summit. 


To collect the conceptions around defining the field for ELDA from the participants in the post- 
event survey, respondents were asked the following open-ended question: 


Given the sessions you attended at the ELDA Summit as well as your own experiences, to you, 


what are the central ideas, issues, and challenges in the domain of Education Leadership Data 
Analytics ? 
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Seventeen of the post-event survey respondents provided a response to this question. 
In reading the responses, five main themes emerged from across the provided responses: 


1. Developing, growing, refining, and incentivizing feedback loops between researchers and 
practitioners in the use of data analytics for instructional improvement. 

2. A crucial area is to build capacity in data analytics with building leaders, teachers, and 
researchers. Multiple respondents noted the central role of building capacity especially at the 
State Education Agency (SEA) level as well as the Local Education Agency level (LEA). 

3. A focus on data privacy, quality, interoperability across data silos, and availability across 
contexts both for data sets and data tools, including FAIR and equitable uses of data. 

4. Acentral issue of how to find faculty to train future administrators in ELDA. 

5. A focus on the human element in education, approaching data and analytics use with humility 
and a collaborative stance to help educators address the issues they have, rather than an 
exclusive focus on questions from researchers. 


SUMMARY & CONCLUSION 


In conclusion, this report on the Education Leadership Data Analytics (ELDA) Summit 2018 lays 
an initial foundation for future efforts in research, policy, and practice at the intersection of 
education leadership, evidence-based improvement cycles, and education data science. First, the 
review of the literature around ELDA shows that the field is emerging, yet researchers and 
practitioners call for a strong need for training, capacity building, networking, and certification 
structures in ELDA, as both practitioners and researchers have few areas to call on for help with 
their work in this domain. Second, the findings from this report indicate that there are some 
professional development offerings at specific overlaps of the ELDA Venn diagram, such as the 
Harvard Strategic Data Project. However, there currently are no strong offerings for practitioners 
or researchers that address the central intersection of ELDA, such as a masters degree or advanced 
certification in ELDA. Currently, practitioners in schools, districts, states, and nations who work 
in ELDA come to the job through very idiosyncratic means, and lack a strong program of 
certification as well as a professional network to help support their work. Third, the results from 
the ELDA Summit 2018 reinforces the current near-term central issues of ELDA, to focus on 
capacity building and practice through integrating research with practical applications to train 
current practitioners in fair and ethical data analysis and data use in an effort to positively inform 
evidence-based improvement cycles. 


In Figure 10 below, we summarize the central skills and competencies for the ELDA field, as an 
integration of the information from the review of the literature, and the responses from the ELDA 
Summit 2018. Here, we use the Bowers (2017) framework for different types of roles in districts 
for ELDA on the left including the Practicing Administrator, Quantitative Analyst, Research 
Specialist, and Education Data Scientist to frame the different facets of the domain. For each area 
within the dashed regions, core skills and competencies could be integrated for each of the four 
job roles. However, the level of integration and depth of required skills and knowledge varies for 
each job role in ELDA. Nevertheless, these main areas in Education Leadership, Evidence-based 
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Improvement Cycles, and Data Science/Data Analytics helps to frame future program, 


certification, and professional development planning in the ELDA domain. 
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Figure 10: Core competencies and skills for Education Leadership Data Analytics (ELDA) 


Our summary of the literature in the ELDA domain provides a means to highlight seven potential 
core competency and skill domains for professional development and training for practitioners and 
researchers in the field. First, these include school building and school system leadership as well 
as community building, such as shared instructional leadership, leadership for learning, and 
culturally responsive leadership (Bowers, Blitz, Modest, Salisbury, & Halverson, 2017; Boyce & 
Bowers, 2018; Khalifa, Gooden, & Davis, 2016; Marks & Printy, 2003; Murphy, Elliott, Goldring, 
& Porter, 2007; Urick & Bowers, 2014). These leadership competencies also include the data 
driven decision-making and program evaluation literature noted above. Second, there is a strong 
need for training in “e-Administration” and database management, as the number and size of 
education datasets in schooling organizations continues to grow (Krumm et al., 2018; Piety et al., 
2014). This is especially important as schooling organizations add additional streams of data, such 
as recommender systems, personalized learning systems, and educational games and apps (Krumm 
et al., 2018; Piety et al., 2014; Piety & Pea, 2018), as well as tracking long-term student outcomes 
and organizational interactions, such as through academic analytics (Agasisti & Bowers, 2017; 
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Siemens, 2013), and intelligent tutors and learning management systems (R. S. Baker & 
Koedinger, 2018; Koedinger, D'Mello, McLaughlin, Pardos, & Rosé, 2015). Third, as 
demonstrated in the responses of the participants in the summit, multiple platforms for coding and 
analytics are the standard. ELDA practitioners should be versed in multiple statistical and data 
science programming languages, such as R, Python, SATA, and SPSS. Additionally, as across the 
data science literature discussed above, data dashboards and user interface design (UX) is a central 
competency, as visualization and interactivity of data analytics and data tools is key for bridging 
the divide between analytics and understanding of stakeholders. 


Fourth, we list data science as a single box, as the field of data science is rapidly developing. 
Nevertheless, given the literature to date on education data science and the intersection with big 
data analytics (Agasisti & Bowers, 2017; Bowers et al., 2016; Krumm et al., 2018; Piety et al., 
2014; Piety & Pea, 2018), inclusion of data science within ELDA is crucial to provide an avenue 
for the innovations from the data sciences to help inform education practice. 


Fifth, data ethics and management of data with a focus on ethical, confidential, and private use of 
data is a core competency. As noted above, to help build the field, FAIR data standards that work 
to provide open and accessible data and tools (Austin et al., 2017), will create part of the 
infrastructure needed for the field to develop. At the same time, ethical use of the data will provide 
safeguards and training on the protection of data privacy, while ensuring that decision algorithms 
in education are open and accessible to understand how machine-informed decisions are being 
made, and the extent that they may be biased or not (Agasisti & Bowers, 2017). 


Sixth, a core competency area is in statistical and machine learning methods to help inform 
predictions, and early warning systems, to understand the data patterns available, and positively 
influence policy and decisions in schools to help support student learning and instruction in 
classrooms. Skills within this domain include education data mining and learning analytics as well 
as inferential statistics (R. S. Baker & Inventado, 2014; R. S. Baker & Yacef, 2009; Krumm et al., 
2018; Piety & Pea, 2018; Siemens, 2013). These skills also include the use of education data to 
make long-term predictions and inform actionable early warning systems and indicators 
(Allensworth, 2013; Allensworth, Nagaoka, & Johnson, 2018; Bowers, Sprott, & Taff, 2013; 
Bowers & Zhou, in press; Davis, Herzog, & Legters, 2013; Mac Iver, 2013). 


Seventh, the final core competency area includes skills on surveying stakeholders as well as 
psychometrics and assessments (Bowers, 2017), while also considering how to evaluate and test 
interventions, such as through evidence-based improvement cycles, as well as cost-benefit 
analysis. 


Overall, while it may not be possible nor even advisable to provide a deep experience in each of 
these core competencies for all four facets of the Practicing Administrator, Quantitative Analyst, 
Research Specialist, and Education Data Scientist, we argue that ELDA researchers and 
practitioners should strive to build these skills over an entire career. It may be the case that initial 
training in ELDA would give entrée into each of the areas, with the profession structuring learning 
opportunities for career professionals at conferences and networking events to help continue to 
build their skills, share their practices, and work in collaboration with partners and stakeholders 
throughout each schooling system. 
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Near-term Recommendations: 


The information collected from this process indicates a strong need for certification and degree 
programs as well as capacity building and professional development offerings in Education 
Leadership Data Analytics, as there appears to be a strong market that is currently underserved in 
this area nationally and globally. Building on the literature review as well as feedback from the 
ELDA Summit 2018, we provide the following near-term recommendations: 


1. 


3. 


Annual or semi-annual ELDA research and practice meetings or conferences. Capacity 
building, training, and networking for current researchers and practitioners is a central 
identified concern in the ELDA field. There is a strong need for conferences and professional 
development opportunities to meet, share practices, and build networks and capacity. 
Additionally, a central recommendation and focus from the field that emerged from the summit 
is the critical role of feedback loops and researcher-practitioner partnerships between 
researchers, practitioners, universities, and schooling systems. We encourage future 
conferences to provide multiple tracks to serve the various needs, including conferences with 
simultaneous research, training, and industry development tracks, bringing each together to 
help bridge between different areas of interest. A useful example of such meetings is the KDD 
annual meeting sponsored by the Association for Computing Machinery's Special Interest 
Group on Knowledge Discovery and Data Mining (ACM SIGKDD), which includes multiple 
tracks that brings together researchers and practitioners of machine learning and data science 


(https://www.kdd.org/). 


Funding for networking and visiting researchers and practitioners. There is a strong need for 
sharing of experiences, research ideas, practical applications, and capacity across the field. We 
recommend that funders work to focus on helping researchers and ELDA practitioners share 
their experiences and data tools by funding teams to be resident in different locations. This 
could take the form of long-term internships and fellowships, as well as more short-term week 
long, or month long training and capacity building. For example, university researchers could 
host three to five people for one or two weeks and focus on a specific topic, and the same could 
take place for school districts where they host a small team to help focus on a specific research 
or practice topic, with both the host and the guests learning and exchanging ideas and tools 
with each other. 


A consortium of like-minded institutions around ELDA. To help continue to build capacity in 
the field, we recommend that like-minded institutions form a consortium to share resources, 
information, and training. This consortium could take many forms, from including universities, 
to also including school districts, as well as NGOs, research groups, and vendors and 
technology companies. Such a consortium could help provide a currently needed central 
resource and deliberative body, as many different data, data management, and data use 
architectures are being deployed to schools. A body that could provide some standardization 
across the industry would benefit the entire ELDA field, as this could help standardize data 
systems, which in turn would make analysis, tool building, and testing easier. 
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4. A national dialogue on curriculum, skills, and core competencies in ELDA. While we provide 
an initial proposed set of core competencies and skills in this report, we encourage more work 
in this area and recommend that ELDA researchers and practitioners work to create a shared 
set of expectations around needed skills and core competencies that would make up a 
curriculum for potential training and certification programs in ELDA. Ensuring an open and 
inclusive dialogue around the core competencies would benefit the ELDA field to ensure that 
training and professional development addresses the needs of schools and the people doing the 
work on the ground to best serve education communities. 
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